Memory usage scheme for performing wavelet processing

ABSTRACT

A system comprising a memory and wavelet processing logic is described. The memory is sized to include lines to store a band of an image and additional lines. The wavelet processing logic comprises a wavelet transform and access logic. The wavelet transform generates coefficients when applied to data in the memory. The access logic reads data from the memory into the line buffers to supply data stored in the memory to the wavelet transform and to store coefficients in the memory, such that after data stored at a first pair of lines is read from memory into the buffers of the access logic. The access logic reuses the first pair of lines to store coefficients generated by the wavelet transform that are associated with a second pair of lines different from the first pair of lines.

FIELD OF THE INVENTION

The present invention relates to the field of compression anddecompression; more particularly, the present invention relates tomemory usage for performing wavelet processing.

BACKGROUND OF THE INVENTION

The new JPEG 2000 decoding standard (ITU-T Rec.T.800/ISO/IEC 154441:2000JPEG 2000 Image Coding System) provides a new coding scheme andcodestream definition for images. Although the JPEG 2000 standard is adecoding standard, the JPEG 2000 specifies encoding and decoding bydefining what a decoder must do. Under the JPEG 2000 Standard, eachimage is divided into one or more rectangular tiles. If there is morethan one tile, the tiling of the image creates tile-components that canbe extracted or decoded independently of each other. Tile-componentscomprise all of the samples of a given component in a tile. An image mayhave multiple components. Each of such components comprises atwo-dimensional array of samples. For example, a color image might havered, green and blue components.

After tiling of an image, the tile-components may be decomposed intodifferent decomposition levels using a wavelet transformation. Thesedecomposition levels contain a number of subbands populated withcoefficients that describe the horizontal and vertical spatial frequencycharacteristics of the original tile-components. The coefficientsprovide frequency information about a local area, rather than across theentire image. That is, a small number of coefficients completelydescribe a single sample. A decomposition level is related to the nextdecomposition level by a spatial factor of two, such that eachsuccessive decomposition level of the subbands has approximately halfthe horizontal resolution and half the vertical resolution of theprevious decomposition level.

Although there are as many coefficients as there are samples, theinformation content tends to be concentrated in just a few coefficients.Through quantization, the information content of a large number ofcoefficients is further reduced. Additional processing by an entropycoder reduces the number of bits required to represent these quantizedcoefficients, sometimes significantly compared to the original image.

The individual subbands of a tile-component are further divided intocode-blocks. These code blocks can be grouped into partitions. Theserectangular arrays of coefficients can be extracted independently. Theindividual bit-planes of the coefficients in a code-block are entropycoded with three coding passes. Each of these coding passes collectscontextual information about the bit-plane compressed image data.

The bit stream compressed image data created from these coding passes isgrouped in layers. Layers are arbitrary groupings of successive codingpasses from code-blocks. Although there is great flexibility inlayering, the premise is that each successive layer contributes to ahigher quality image. Subband coefficients at each resolution level arepartitioned into rectangular areas called precincts.

Packets are a fundamental unit of the compressed codestream. A packetcontains compressed image data from one layer of a precinct of oneresolution level of one tile-component. These packets are placed in adefined order in the codestream.

The codestream relating to a tile, organized in packets, are arranged inone, or more, tile-parts. A tile-part header, comprised of a series ofmarkers and marker segments, or tags, contains information about thevarious mechanisms and coding styles that are needed to locate, extract,decode, and reconstruct every tile-component. At the beginning of theentire codestream is a main header, comprised of markers and markersegments, that offers similar information as well as information aboutthe original image.

The codestream is optionally wrapped in a file format that allowsapplications to interpret the meaning of, and other information about,the image. The file format may contain data besides the codestream.

The decoding of a JPEG 2000 codestream is performed by reversing theorder of the encoding steps. FIG. 1 is a block diagram of the JPEG 2000standard decoding scheme that operates on a compressed image datacodestream. Referring to FIG. 1, a bitstream initially is received bydata ordering block 101 that regroups layers and subband coefficients.Arithmetic coder 102 uses contextual information collected duringencoding about the bit-plane compressed image data, and its internalstate, to decode a compressed bit stream.

After arithmetic decoding, the coefficients undergo bit modeling incoefficient bit modeling block 103. Next, the codestream is quantized byquantization block 104, which may be quantizing based on a region ofinterest (ROI) as indicated by ROI block 105. After quantization, aninverse transform is applied to the remaining coefficients via transformblock 106, followed by DC and optional component transform block 107.This results in generation of a reconstructed image.

The JPEG2000 standard leaves many choices to implementers.

SUMMARY OF THE INVENTION

A system comprising a memory and wavelet processing logic is described.The memory is sized to include lines to store a band of an image andadditional lines. The wavelet processing logic comprises a wavelettransform and access logic. The wavelet transform generates coefficientswhen applied to data in the memory. The access logic reads data from thememory into the line buffers to supply data stored in the memory to thewavelet transform and to store coefficients in the memory, such thatafter data stored at a first pair of lines is read from memory into thebuffers of the access logic. The access logic reuses the first pair oflines to store coefficients generated by the wavelet transform that areassociated with a second pair of lines different from the first pair oflines.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be understood more fully from the detaileddescription given below and from the accompanying drawings of variousembodiments of the invention, which, however, should not be taken tolimit the invention to the specific embodiments, but are for explanationand understanding only.

FIG. 1 is a block diagram of the JPEG 2000 standard decoding scheme.

FIG. 2 illustrates one embodiment of an organization for an image inmemory.

FIGS. 3A-F illustrate a transform memory organization for various levelsdepicting conceptually how coefficients may be stored for the forward(Figures A-C) and inverse (FIGS. 3D-F) transforms.

FIGS. 4A and B illustrate embodiments of a single memory where the inputimage data and the various decomposition levels of the image data can bestored during forward and inverse transforms, respectively.

FIG. 5 illustrates one embodiment of the process of handling the inputdata.

FIG. 6A illustrates a system having a progression order conversionparser.

FIG. 6B illustrates a progression converter converting from a resolutionprogressive codestream to a quality progressive codestream.

FIG. 7A shows multiple ways to convert a codestream from one progressionorder to another.

FIG. 7B shows one embodiment of simplified conversion paths to convert acodestream from one progression order to another.

FIG. 8 illustrates one embodiment of a process for performingprogression order conversion.

FIG. 9 illustrates a decoder that selects portions of a codestream basedon sideband information.

FIG. 10 is a flow diagram of a process for using layers when decoding.

FIG. 11 is a flow diagram of one embodiment of an editing process.

FIG. 12 illustrates a bell-shaped curve of a range of values that arequantized to a particular value.

FIG. 13 is a flow diagram of one embodiment of a process to reduceflicker.

FIG. 14 illustrates one embodiment of an encoder (or portion thereof)that performs the quantization to reduce flicker.

FIG. 15A illustrates a process for performing rate control.

FIG. 15B illustrates an exemplary number of layers that may be subjectedto first and second passes.

FIG. 16 illustrates one embodiment of the process for accessing thegroupings of tile parts.

FIGS. 17 and 18 illustrate quantizers for one component for a threelevel 5,3 transform.

FIG. 19 illustrates an example of HVS weighted quantization.

FIG. 20 is a block diagram of one embodiment of a computer system.

FIG. 21 illustrates an example progression with tile parts for a singleserver.

FIG. 22 illustrates an example of layering for a 5,3 irreversibletransform.

FIG. 23 illustrates an example in which transform has 5 levels and thedata is divided up into layers 0-3.

FIG. 24 illustrates one example of a situation in which flicker may beavoided in which values in first and third frames are used to set thevalue in the second frame.

FIG. 25 is a block diagram of a prior art decoding process that includescolor management.

FIG. 26 illustrates one embodiment of a non-preferred camera encoder.

FIG. 27 illustrates one embodiment of a simpler camera encoder.

FIG. 28 is a flow diagram of one embodiment of a process for applying aninverse transform with clipping on partially transformed coefficients.

DETAILED DESCRIPTION OF THE PRESENT INVENTION

Improvements to compression and decompression schemes are described. Itis a purpose of the techniques and implementations described herein touse choices in JPEG 2000 to make high speed, low cost, low memory and/orfeature rich implementations.

In the following description, numerous details are set forth in order toprovide a thorough explanation of the present invention. It will beapparent, however, to one skilled in the art, that the present inventionmay be practiced without these specific details. In other instances,well-known structures and devices are shown in block diagram form,rather than in detail, in order to avoid obscuring the presentinvention.

Some portions of the detailed descriptions which follow are presented interms of algorithms and symbolic representations of operations on databits within a computer memory. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of steps leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, symbols, characters,terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the following discussion,it is appreciated that throughout the description, discussions utilizingterms such as “processing” or “computing” or “calculating” or“determining” or “displaying” or the like, refer to the action andprocesses of a computer system, or similar electronic computing device,that manipulates and transforms data represented as physical(electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission or display devices.

The present invention also relates to apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, or it may comprise a general purpose computerselectively activated or reconfigured by a computer program stored inthe computer. Such a computer program may be stored in a computerreadable storage medium, such as, but is not limited to, any type ofdisk including floppy disks, optical disks, CD-ROMs, andmagnetic-optical disks, read-only memories (ROMs), random accessmemories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any typeof media suitable for storing electronic instructions, and each coupledto a computer system bus.

The algorithms and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general purposesystems may be used with programs in accordance with the teachingsherein, or it may prove convenient to construct more specializedapparatus to perform the required method steps. The required structurefor a variety of these systems will appear from the description below.In addition, the present invention is not described with reference toany particular programming language. It will be appreciated that avariety of programming languages may be used to implement the teachingsof the invention as described herein.

A machine-readable medium includes any mechanism for storing ortransmitting information in a form readable by a machine (e.g., acomputer). For example, a machine-readable medium includes read onlymemory (“ROM”); random access memory (“RAM”); magnetic disk storagemedia; optical storage media; flash memory devices; electrical, optical,acoustical or other form of propagated signals (e.g., carrier waves,infrared signals, digital signals, etc.); etc.

Overview

The following descriptions relate to implementations or novel ways totake advantage of the flexibility of JPEG 2000 or other coding schemeswith similar features.

Memory Usage for Low Memory and Fast Burst Access

FIG. 2 shows one embodiment of an organization for an image in memory201. Referring to FIG. 2, only the “tile height” raster lines, or a bandof the image, are in memory 201, not the whole image. Thus, the amountof an image in memory 201 is equal to the image width multiplied by thetile height. Inside the band of the image is at least one tile, such astile 210.

The wavelet transform processing logic 202 includes memory access logic202A to read data from and store data to memory 201 to enable wavelettransform 202B to be applied to the data (image data or coefficientsdepending on the level of coefficient). Wavelet processing logic 202 maycomprise hardware, software or a combination of both.

In one embodiment, access logic 202A accesses the tile with fourparameters: a pointer or index to the start of the tile in memory, thewidth of the tile, the height of the tile, and the line offset to getfrom the start of one line to another (the image width). Alternatively,access logic 202A accesses memory 201 using a pointer or index to theend of the tile instead of the width of the tile.

In one embodiment, in order to access for each line of a tile or aportion of a line of an image to perform some function F, the followingprocess may be used.

line=start

for y=0 to tile_height−1

-   -   for x=0 to tile_width−1        -   perform function F with line[x]    -   line=line+line_offset        One of the functions F may include applying a wavelet transform        on pairs of lines. Also another function F may be a DC level        shift, multiple component transform.

Such a process would be performed by processing logic that may comprisehardware (e.g., dedicated logic, circuitry, etc.), software (such as isrun on a general purpose computer system or a dedicated machine), or acombination of both.

In one embodiment, coefficients from a subband are accessed using asimilar scheme with a starting point, width, height and line offset.Because rows of coefficients are stored together in memory, rows may beaccessed efficiently when stored in cache, burst accessible memories ormemories that are wider than one coefficient.

FIGS. 3A-C show a transform memory organization for various levelsdepicting conceptually how coefficients may be stored. All LH, HL and HHcoefficients (using the nominclature of ITU-T Rec.T.800/IS0/IEC154441:2000 JPEG 2000 Image Coding System) are coded. These coefficientsare represented by dotted lines in FIGS. 3B and C. Input lines of inputtile 301 and LL coefficients (shown as solid lines in FIGS. 3B and 3C insuccessive levels) only need to be stored temporarily while computingthe transform with the exception of the final transform level's LLcoefficients which are coded. When a transform is used that does thehorizontal and vertical transforms in one pass and uses line buffers,once a pair of input rows has been completely read (input lines or LLcoefficients), the space used by lines can be reused.

FIGS. 3A-C show input tile 301, level 1 (L1)(302) and level 2 (L2)(303)memory areas aligned with an offset to indicate how reuse might beaccomplished in one embodiment. The addition of two rows, rows 312 and313, to the memory space used to hold input tile 301, is needed togenerate the L1 coefficients when reusing the memory for input tile 301for L1 coefficients. The addition of four rows, rows 341-342, to thememory space used to hold the L1 coefficients is needed to generate theL2 coefficients when reusing the memory storing the L1 coefficients forL2 coefficients. (Note that there are two rows between rows 341 and 342that are wasted space.) The additional lines are preferably behind thedirection the wavelet transform is being applied to the information inthe memory.

More specifically, a pair of input rows input tile 301 corresponds toone row of each of LL, LH, HL and HH coefficients at level 1, resultingfrom the application of a transform to two different lines and theresults of applying the wavelet transform being written into lines ofthe memory. For example, the results of applying a wavelet transform toinput rows 310 and 311 are the coefficients in portions of rows 312 and313 of L1 coefficients (302). For example, LL coefficients 321 of row312 corresponds to the LL coefficients (solid line) of level 1, HLcoefficients 322 of row 312 corresponds to the HL coefficients of level1, LH portion 323 of row 313 corresponds to the LH coefficients of level1, and HH portion 324 corresponds to the HH coefficients of level 1.Note that the level 1 coefficients from the first two input lines arestored in two extra rows at the top of the memory with the remaininglevel 1 coefficients being written into the locations storing the dataof input tile 301 to reuse the memory. The width and height for eachtype of coefficient (e.g., LH, HL, HH) for each subband of level 1coefficients is half that of input tile 301. The line offset from the LLrow to the next LL row for level 1 (e.g., the distance from row 312 torow 314 in FIG. 3B) is twice the tile width (since each coefficient rowis from an area corresponding to two lines).

Similarly, the results of applying the wavelet transform to two rows ofLL coefficients at level 1 (solid lines) are the coefficients in tworows namely LL coefficients (331), LH coefficients (332), HLcoefficients (333) and HH coefficients (334) at level 2. The width andheight for level 2 coefficients is a quarter that of input tile 301. Theline offset for level 2 is four times the tile width (since eachcoefficient row is from an area corresponding to two level 1 LL rows orfour input lines). Thus, four extra lines of memory are needed to usethe same memory that is storing the input tile to store the L2coefficients. Note that if a third decomposition level was beingperformed, an additional 8 lines would be needed. Thus, in this example,a total of 14 extra lines are needed to enable reuse of the memory thatstores an input time and has two levels of decomposition appliedthereto. A general formula may be used to determine the number of extralines is as follows:2^((maxlevel+1))−2.

To access subbands, such as the LL, LH, HL and HH subbands, only astarting pointer and the offset between rows/lines are necessary. Theheight and width are also needed to know when to stop when accessing atile.

As the number of decomposition levels increases, some rows at the bottomof memory become unused. That is, the lines of memory below the L1coefficients after the first decomposition level become unused, thelines of memory below the L2 coefficients after the second decompositionlevel become unused, etc. In one embodiment, this extra space may bereused.

FIGS. 3D-3F illustrate the corresponding inverse transform memory usagein which additional lines store the results of applying an inversetransform and those additional lines are in the memory behind thedirection the inverse transform is being performed.

FIG. 4A shows one embodiment of a single memory where the input and thevarious levels can be stored during application of a forward transform.Referring to FIG. 4A, locations for the input tile, level 1coefficients, level 2 coefficients, and level 3 coefficients is shownwith the added 2,4 and 8 lines respectively. FIG. 4B shows a similarsingle memory embodiment where the input coefficients of various levelsof the transform can be stored along with the output during applicationof an inverse transform.

Table 1 shows the amount of memory required for various transform levelsfor a 256×256 tile for separate memories and reused memory.

TABLE 1 Separate memory level (bytes) reused memory (bytes) 1 256 × 256= 65,536  2 × 256 = 512 2 128 × 128 = 16,384  4 × 256 = 1,024 3  64 × 64= 4,096  8 × 256 = 2,048 4  32 × 32 = 1,024 16 × 256 = 4,096 5  16 × 16= 256 32 × 256 = 8,192 6  8 × 8 = 64 64 × 256 = 16,384

For reused memory, the amount listed is the additional new memory usedfor that level. For this example, reusing memory for levels 1, 2 and 3saves memory. Level 4 may use a separate memory.

The memory for levels 4,5 and 6 could be placed in a single memory afterlevel 3 has been generated or in a completely different and separatememory. The amount of memory necessary is 38×32, which is less than5×256. Because there are two unused lines after generating the level 1coefficients (i.e., the memory that stored the last two lines of inputdata), a small memory savings can be achieved by letting the levels 4, 5and 6 reuse these two lines. This is particularly important because thenumber of additional lines for levels 4, 5, and 6 is 16, 32 and 64, andthe extra space between the lines will be twice as far and half as wideas the level before.

In one embodiment, coefficients from levels 4, 5, and 6 are packed in asmaller memory structure, such as storage area 450 in FIG. 4. Referringto FIG. 4, the level 4 coefficients are stored in an area having aheight equal to the tile height divided by 8 (2³ where 3 corresponds tothe number of levels) and a width equal to the tile width w divided by 8(2³ where 3 corresponds to the number of levels previously storedelsewhere). An additional two lines 451 are all that is needed to storelevel 5 coefficients in the same necessary storage area. Similarly, anadditional four lines is all that is necessary to accommodate using thismemory storage area for the level 6 coefficients. Note that no lines areskipped when storing the coefficients. In one embodiment in which a256×256 tile is being processed, the extra 5 lines at the bottom ofstorage area 430, two lines 421 and approximately 4.75 lines 422 areused to accommodate storage area 450. As shown, the approximate by threelines 422 represent allocated memory or in addition to that necessary tostore the input tile. In this manner, the storage area for the inputtile is almost completely reused.

In one embodiment, to use a very little, or potentially minimum, memory,level 6 is stored separately from levels 4 and 5. However, this onlysaves 64 bytes of memory.

A memory a little smaller than 273×256 can hold all the transformcoefficients for a 256×256 tile. This is less than 7% more than a truein-place memory organization. Unlike an in-place memory organization,extra copies are avoided while simultaneously keeping the rows packedtogether for fast access.

Table 2 shows another example of using separate versus reused memory for128×128 tiles. For this size, the first three transform levels can reusememory in a 142×128 buffer.

TABLE 2 Separate memory level (bytes) reused memory (bytes) 1 128 × 128= 16,384 2 × 128 = 256 2  64 × 64 = 4,096 4 × 128 = 512 3  32 × 32 =1,024 8 × 128 = 1024

In one embodiment, a decision to use in-place memory or new memory is afunction of tile height and transform level. Such a decision may bebased on the following:

-   -   if tile height>2^((3*level−2)), then use in-place method    -   if tile height=2^((3*level−2)), then either may be used    -   if tile height<2^((3*level−2)), then use new memory        To illustrate the application of the decision, Table 3 below:

TABLE 3 level 2{circumflex over ( )}(3*level-2) 1 2 2 16 3 128 4 1024 58192

In some applications, adapting the memory organization to the tileheight is inconvenient. A single fixed memory organization can be used.Tile sizes smaller than 128×128 typically result in bad compressionperformance, so would typically not be used. While tile sizes biggerthan 1K×1K can be used for very large images, this does notsignificantly improve compression and the large amount of memoryrequired would typically be burdensome. Therefore, assuming a tileheight between 128 and 1024 inclusive and using in-place memory for 3levels of the transform is a good heuristic.

Decoding is similar in that the results of applying an inverse transformare written ahead of where the decoding processing logic is reading,with the only notable difference being that the start is from thehighest level to the lowest level, such as level 6 to level 1 in theexample above. In such a case, the input tile ends up at the top of thememory structure. The extra lines to accommodate the memory reuse are indecreasing order. For example, using the structure of FIG. 4B, 8 lineswould be necessary to create the L2 coefficients from the L3coefficients, 4 extra lines would be necessary to create the L1coefficients from the L2 coefficients and 2 extra lines would benecessary to create the input tile from the L1 coefficients.

In one embodiment, to handle input tile data, a color conversion may beperformed on the data prior to encoding. FIG. 5 illustrates oneembodiment of the process of handling the input data. Referring to FIG.5, color input pixels are received in raster order. These color pixelsmay be in RGB, YCrCb, CMY, CMYK, grayscale, etc. The color input pixelsmay be stored as tiles in a memory, such as memory 501, by band (orother forms).

Pixels from storage 501 or received directly form the input undergocolor conversion and/or level shifting, with the resulting outputs beingstored in one coefficient buffers 502 ₁-502 _(N). That is, once thecolor conversion has been completed on each tile, it is stored in one ofthe coefficient buffers 502 ₁-502 _(N), and then the next tile can beprocessed. In one embodiment, there is one coefficient buffer for eachcomponent.

Coefficient buffers 502 ₁-502 _(N) are used by the transform in themanner described above to perform the wavelet transform while reusingmemory. Thus, coefficient buffers 502 ₁-502 _(N) are both input andoutput to wavelet transform.

After the transform is applied to coefficient buffers 502 ₁-502 _(N),the context model 503 and entropy coder 505 can perform furthercompression processing on the already transformed data. The coded datais buffered in coded data memory 505.

While performing the further compression processing on one tile, thetransform may be applied to another tile. Similarly, any or all theoperations may be performed on multiple tiles at the same time.

Progression Order Conversion

In the JPEG2000 standard, data in a compressed codestream can be storedin one of the five progression orders. The progression order can changeat different points in the codestream. The order is defined by embedded“for layers” on layers, precincts, resolution, and components.

Five progression orders are described in the standard in Table A-16 ofthe JPEG 2000 standard. They are layer-resolution-component-positionprogression (LRCP), resolution-layer-component-position progression(RLCP), resolution-position-component-layer progression (RPCL),position-component-resolution-layer progression (PCRL),component-position-resolution-layer progression (CPRL).

The order may be defined in the COD or POC markers of the JPEG 2000standard. The Coding style default (COD) marker is defined by the JPEG2000 standard and describes the coding style, number of decompositionlevels, and layering that is the default used for compressing allcomponents of an image (if in the main header) or a tile (if in atile-part header). The Progression order change (POC) marker describesthe bounds and progression order for any progression order other thanthat specified in the COD marker segments in the codestream. The PacketLength Main Header (PLM) indicates a list of packet lengths intile-parts for every tile part in order and the Packet Length, Tile-partheader (PLT) indicates tile packet lengths in a tile-part and indicateswhere the data is in the codestream.

The JPEG 2000 standard in section B.12 only specifies how packets ofcompress data are formed for a given progression order. It does notdescribe how data should be converted from one progression order toanother progression order.

In one embodiment, a progression order converting parser converts acodestream to a desired progression order based on the user inputwithout decoding the data and then encoding it again. FIG. 6Aillustrates a system having such a parser. Referring to FIG. 6A, parser601 receives requests from a client for a particular progression order.The client may be viewing a web page and selects a particular link. Inresponse to the request, parser 601 accesses server 602 to obtain thecodestream associated with full image 603 from memory 604 and convertsthe codestream into a different progression order based on the request.The request indicates the progression order by using an optional command(e.g., RL2L (Resolution-layer progression to Layer Progression)). Theprogression order that is described may be based on layer, resolution,component, precinct, or tile.

FIG. 6B illustrates the progression converter converting from a layerprogressive codestream (LRCP) to a resolution progressive (RLCP)codestream. The progression orders map directly to each other.

FIG. 7A shows multiple ways to convert a codestream from one progressionorder to another. Referring to FIG. 7A, each of the five progressions(LRCP, RLCP, RPCL, CPRL, and PCRL) are shown with paths to each of theothers, such that all progressions are shown. In one embodiment, theparser causes all conversions to go through the layer progression firstand then to a selected conversion. FIG. 7B shows one embodiment of suchsimplified conversion paths in which the number of required mappings isreduced from 10 (as in FIG. 7A) to 4. However, any one of the fiveprogression orders could be used as the one to which all are convertedbefore arriving at the selected order. The conversion techniquedescribed herein simplifies source codes in that the number of lines ofsource code is much less than the multiple ways of conversion. Thisresults in less debug time and fewer memory and run-time variables.

To perform the conversion, the order of the packets in the codestreammust be reordered. The packets are labeled by their sequential order inthe codestream. Markers may indicate the starting point of the data, thelength of the data (or alternatively the endpoint of the data) and howthe data should be handled. For example, the indication of how the datais to be handled may indicate whether the data is to be deleted, whetherthe data is to be truncated, or some other operation to be performed onthe data. Such handling information may also come from rate distortioninformation, such as may be provided in a PLT/PLM and/or the PPT/PPMmarker sets of the JPEG 2000 standard. In this manner, the codestreammay be truncated without changing the packet header.

In one embodiment, a list, array, or other structure (such as reorderingstructure 601A) is built by indicating the portion of data in eachpacket. Using this structure, the packets may be reordered.

FIG. 8 illustrates one embodiment of a process for performingprogression order conversion. The process is performed by processinglogic that may comprise hardware (e.g., dedicated logic, circuitry,etc.), software (such as is run by, for example, a general purposecomputer or dedicated machine), or a combination of both.

Referring to FIG. 8, the process begins by processing logic building alist from headers in the packets (processing block 801) and optionallymarking list items “delete” for quantization (processing block 802).Next, processing logic reorders the list to map the original progressionto a desired progression (including handling input and output withprogressions specified with POC markers (bounds on the progressionorder) (processing block 803). Thereafter, processing logic outputscoded data based on reordered list (processing block 804).

Therefore, the combination of re-ordering and parsing allowsspecification of the desired ordering and resolution, quality, etc.

A Progression Order Conversion Example

The following is an example showing how packets are arranged in acodestream. The codestream was formed based on 2 components, 2 layers, 3decomposition levels, and layer progression.

Table 4 shows the packet order, length and association index of packetsin the example. The packet order column shows the sequential order ofpackets placed in a codestream. The length indicates the length of thepackets. The association index shows the resolution, layer, component,and precinct of the packet.

For example, packet[0] is the first packet in the codestream after thefirst tile header. It has a length of 589 bytes. Association indexRwLxCyPz indicates the packet belongs to resolution w, layer x,component y and precinct z.

TABLE 4 Packet order Length Association Index packet[0] length = 589R0L0C0P0 packet[1] length = 589 R0L0C1P0 packet[2] length = 924 R1L0C0P0packet[3] length = 924 R1L0C1P0 packet[4] length = 1602 R2L0C0P0packet[5] length = 1602 R2L0C1P0 packet[6] length = 733 R3L0C0P0packet[7] length = 733 R3L0C0P0 packet[8] length = 535 R0L1C0P0packet[9] length = 535 R0L1C1P0 packet[10] length = 1523 R1L1C0P0packet[11] length = 1523 R1L1C1P0 packet[12] length = 5422 R2L1C0P0packet[13] length = 5422 R2L1C1P0 packet[14] length = 16468 R3L1C0P0packet[15] length = 16468 R3L1C1P0

In this codestream, packets are grouped based on the layer in which theyreside. The first 8 packets belong to Layer 0. The following 8 packetsbelong to Layer 1.

Using the conversion process described herein, the above codestream isconverted to resolution layer progression. The following shows how theabove packets are re-ordered.

After the layer progressive codestream is converted to resolutionprogression, in the new codestream, packets are grouped based onresolution. Such a grouping is shown in Table 5. The first 4 packetsbelong to the resolution 0, the next 4 packets to resolution 1, and soon.

TABLE 5 Previous Packet order Packet order Length Association Index 0packet[0] length = 589 R0L0C0P0 1 packet[1] length = 589 R0L0C1P0 8packet[2] length = 535 R0L1C0P0 9 packet[3] length = 535 R0L1C1P0 2packet[4] length = 924 R1L0C0P0 3 packet[5] length = 924 R1L0C1P0 10packet[6] length = 1523 R1L1C0P0 11 packet[7] length = 1523 R1L1C1P0 4packet[8] length = 1602 R2L0C0P0 5 packet[9] length = 1602 R2L0C1P0 12packet[10] length = 5422 R2L1C0P0 13 packet[11] length = 5422 R2L1C1P0 6packet[12] length = 733 R3L0C0P0 7 packet[13] length = 733 R3L0C1P0 14packet[14] length = 16468 R3L1C0P0 15 packet[15] length = 16468 R3L1C1P0One Embodiment of a Conversion Algorithm

Resolution to Layer Progression n = 0; for(1=0;1<layer;1++){for(r=0;r<resolution+1;r++){ for(c=0;c<component;c++){ new_packet[n]=old_packet[1*component + r*layer*com- ponent + c]; n++; } } } Layer toResolution Progression n = 0; for(r=0;r<resolution+1;r++){for(1=0;1<layer;1++){ for(c=0;c<component;c++){ new_packet[n]=old_packet[r*component + 1*(resolution+1)*component + c]; n++; } } }where layer = the number of layers in a codestream, resolution = thenumber of decomposition levels in a codestream, and component = thenumber of components in a codestreamData Hiding (Sideband Information) in JPEG2000 Coding

Bit hiding allows sideband information to be transmitted withoutincreasing the file size. Sideband information that does increase filesize but does not break naive decoders might also be valuable (althoughthe COM marker defined by the JPEG 2000 standard might be used instead).

Some marker segments, packet headers and packets are padded out to thenearest byte. Examples of the JPEG 2000 marker segments include PPM,PPT, PLM, and PLT. In addition, some marker segments can be longer thanthey need to be including QCD, QCC, and POC. In all of these cases, thepadded data values are not defined.

Several proprietary coding schemes could use this semi-randomly locatedundefined data to provide a number of important types of informationincluding, but not limited to, decoding and filtering hints, ownership,segmentation hints, and so on. A hint might include an index to aparticular enhancement scheme. For example, if it is known that an imageis mostly text, a value may be sent that indicates that a firstpost-processing filter is to be used. On the other hand, if the area ismostly a graphic image, then a value may be sent that indicates that asecond post-processing filter is to be used.

The following are places where bits may be hidden or sidebandinformation may be stored in the codestream.

-   -   arithmetic coder (AC) termination (without predictable        termination)    -   end of packet header rounding to byte    -   after last packet, before next tile    -   tag tree construction by not always using minimum    -   packet header Lblock signalling    -   LSB parity for codeblocks (refinement pass only, cleanup pass        only, all)    -   QCD, QCC extra subbands, POC.

For example, with respect to hiding data using AC termination, 0 to 7bits are provided, at least, everytime the coder is terminated. However,this could be extended for a few bytes. These extra bits and bytes maybe used for sending extra information.

With respect to each packet header, the end of a packet header isrounded to a byte boundary. Therefore, there may be 1 to 7 bits that maybe available for sending extra information at times when rounding wouldhave been necessary. Similarly, each packet is rounded to a byteboundary, thereby providing 1 to 7 bits (assuming that rounding wouldhave been necessary). Also the last packet in a tile-part can beextended a few bytes. These extra bytes may be used to send additionalinformation.

The length of the compressed data for a code-block can be given in thepacket header with a non-minimum representation. The choice ofrepresentation (e.g., a non-minimum representation) could be used forindicating other information.

With respect to tag tree data hiding, packet headers of the JPEG 2000standard use tag trees for coding first inclusion and zero bitplaneinformation. When there are multiple codeblocks, tag trees are like aquadtree of minimum values. For example, in the case of 16 codeblocks ina 4×4 arrangement in a packet, the arrangement may be as follows:${\begin{matrix}10 & {\quad 7} & 12 & 15 \\{\quad 3} & 20 & 21 & {\quad 5} \\81 & 45 & {\quad 5} & {\quad 9} \\18 & {\quad 8} & 12 & 24\end{matrix}\quad}\quad$An example tag tree, which is minimal for the 4×4 arrangement above isas follows: $\quad{\begin{matrix}3 \\\quad \\\quad \\\quad\end{matrix}\quad\begin{matrix}0 & 2 \\5 & 2 \\\quad & \quad \\\quad & \quad\end{matrix}\quad\begin{matrix}{\quad 7} & {\quad 4} & {\quad 7} & 10 \\{\quad 0} & 17 & 16 & {\quad 0} \\73 & 37 & {\quad 0} & {\quad 4} \\10 & {\quad 0} & {\quad 7} & 19\end{matrix}}\quad$in which “3” is added to every codeblock's value, and “0”, “2”, “5” and“2” are each added to the 4 corresponding codeblocks. Finally, there isone value per codeblock. That is, the minimal tag tree is created bytaking the first 2×2 group in the 4×4 arrangement above and look atminimum value is out of the four values. In this case, for the 2×2 block${\quad\begin{matrix}10 & {\quad 7} \\{\quad 3} & 20\end{matrix}\quad}\quad$the minimum value is 3. This is then performed on the other 2×2 blocks.Then these identified minimum values are evaluated again to determinetheir minimum, which would be “3” in the example. Then the minimum valueis subtracted from the four minimum values to create the following${\quad\begin{matrix}0 & 2 \\5 & 2\end{matrix}\quad}\quad$Then, for the remaining numbers in the 4×4, the number 3 is subtractedfrom each value along with the value in the 2×2 that corresponds to theparticular value in the 4×4 arrangement, thereby resulting in the tagtree above.The first row adds up as follows:

-   10=3+0+7-   7=3+0+4-   12=3+2+7-   15=3+2+10

A variable length code may be used that efficiently represents smallnumbers.

An example of a tag tree that is not minimal is as follows:$\quad{\begin{matrix}2 \\\quad \\\quad \\\quad\end{matrix}\quad\begin{matrix}1 & 3 \\6 & 3 \\\quad & \quad \\\quad & \quad\end{matrix}\quad\begin{matrix}{\quad 7} & {\quad 4} & {\quad 7} & 10 \\{\quad 0} & 17 & 16 & {\quad 0} \\73 & 37 & {\quad 0} & {\quad 4} \\10 & {\quad 0} & {\quad 7} & 19\end{matrix}}\quad$(Note that representing “3”, “0”, “2”, “5” and “2” might use lessbitstream data than “2”, “1”, “3”, “6” and “3”.)

Once a tag tree representation has been made, a determination can bemade as to whether the representation is minimal or not based on whetherthere is a zero in the 2×2 block. Therefore, this information is hidden.For example, the 1 bit block represents the 1 in the 2×2 block aboveindicates it is not part of a minimal tag tree, but can be used toconvey some particular information to a decoder. Likewise if a 2 was theminimal value in the 2×2 block, such a fact may convey differentinformation to a decoder.

The JPEG 2000 POC, QCD, and QCC markers can have redundant entries. Itis as if the codestream were quantized and the markers were notrewritten. For example, the QCD and QCC markers have values for a numberof subbands specified by the syntax of the marker. If there are fewersubbands actually coded in the bitstream, data may be hidden in thevalues used for the missing subbands. The redundant entries may bereplaced and used for hidden or sideband information.

The hidden or sideband information may include post-processing hints(such as, for example, sharpen this tile with a specified filter orstrength, or smooth, or perform optical character recognition (OCR) onthis region, etc.), decoding hints, security (such as, for example, anencryption key for decoding the remainder of the image or another image,etc.) codestream identification (such as, for example, labeling POTUS asthe originator of the file, etc.) and/or other information.

Use of Layers When Encoding

Layers are part of the JPEG standard. In one embodiment, sidebandinformation, possibly in a COM marker, is used by the decoder to allowselecting of layers during decoding. The sideband information may beused to select layers for postcompression quantization to meetrate/distortion targets for different viewing distances, differentresolutions, different regions of interest, different frequency contentfor analysis (e.g., finding edges of text).

In one embodiment, the layers are predefined based on rate. For example,the first layer represents a 1-bit per pixel image, while the secondlayer represents a 2-bit per pixel image, etc. Therefore, the layers runfrom the lowest quality to the highest quality. Likewise, target ratescan be met for lower resolutions as well.

The sideband information may be stored in a marker segment of thecodestream. In one embodiment, the JPEG 2000 comment (COM) marker isused to provide information about the layers. Specifically, the COMmarker may be used to indicate the number of bytes for each resolutionand/or rate across the entire image or a relative number of bytes foreach additional layer. Table 6 indicates each layer and its resolutionin the number of bytes across the tile in an image. Such a table mayhave distortion values instead.

TABLE 6 lev = 0 layer = 0 comp = 0 bytes = 529 lev = 0 layer = 0 comp =1 bytes = 555 lev = 0 layer = 0 comp = 2 bytes = 493 lev = 0 layer = 1comp = 0 bytes = 129 lev = 0 layer = 1 comp = 1 bytes = 130 lev = 0layer = 1 comp = 2 bytes = 123 lev = 0 layer = 2 comp = 0 bytes = 7 lev= 0 layer = 2 comp = 1 bytes = 8 lev = 0 layer = 2 comp = 2 bytes = 12lev = 0 layer = 3 comp = 0 bytes = 1 lev = 0 layer = 3 comp = 1 bytes =1 lev = 0 layer = 3 comp = 2 bytes = 129 lev = 1 layer = 0 comp = 0bytes = 705 lev = 1 layer = 0 comp = 1 bytes = 898 lev = 1 layer = 0comp = 2 bytes = 712 lev = 1 layer = 1 comp = 0 bytes = 146 lev = 1layer = 1 comp = 1 bytes = 114 lev = 1 layer = 1 comp = 2 bytes = 116lev = 1 layer = 2 comp = 0 bytes = 224 lev = 1 layer = 2 comp = 1 bytes= 250 lev = 1 layer = 2 comp = 2 bytes = 263 lev = 1 layer = 3 comp = 0bytes = 201 lev = 1 layer = 3 comp = 1 bytes = 212 lev = 1 layer = 3comp = 2 bytes = 200 lev = 2 layer = 0 comp = 0 bytes = 889 lev = 2layer = 0 comp = 1 bytes = 1332 lev = 2 layer = 0 comp = 2 bytes = 1048lev = 2 layer = 1 comp = 0 bytes = 240 lev = 2 layer = 1 comp = 1 bytes= 329 lev = 2 layer = 1 comp = 2 bytes = 328 lev = 2 layer = 2 comp = 0bytes = 599 lev = 2 layer = 2 comp = 1 bytes = 767 lev = 2 layer = 2comp = 2 bytes = 725 lev = 2 layer = 3 comp = 0 bytes = 335 lev = 2layer = 3 comp = 1 bytes = 396 lev = 2 layer = 3 comp = 2 bytes = 420lev = 3 layer = 0 comp = 0 bytes = 1 lev = 3 layer = 0 comp = 1 bytes =395 lev = 3 layer = 0 comp = 2 bytes = 402 lev = 3 layer = 1 comp = 0bytes = 251 lev = 3 layer = 1 comp = 1 bytes = 450 lev = 3 layer = 1comp = 2 bytes = 562 lev = 3 layer = 2 comp = 0 bytes = 525 lev = 3layer = 2 comp = 1 bytes = 990 lev = 3 layer = 2 comp = 2 bytes = 1313lev = 3 layer = 3 comp = 0 bytes = 1214 lev = 3 layer = 3 comp = 1 bytes= 1798 lev = 3 layer = 3 comp = 2 bytes = 2585

In another embodiment, the ordering could be by layer. Thus, theinformation above is consolidated for each level (not segregated bylevel or component), as shown below:

-   Ordering by layer=0 bytes=7959 bitrate=0.971558 PSNR=30.7785-   Ordering by layer=1 bytes=10877 bitrate=1.327759 PSNR=32.0779-   Ordering by layer=2 bytes=16560 bitrate=2.021484 PSNR=35.7321

Distortion by layers can be based on PSNR. For example,

-   layer=0 PSNR=30.7785-   layer=1 PSNR=32.0779-   layer=2 PSNR=35.7321

In an alternative embodiment, such information may be hidden in thecodestream as described above. The information may be used to controlrate distortion.

In another embodiment, the layers may be predefined for a particularviewing distance. In such a case, the data is divided into layers fromthe highest frequency, lowest resolution to the lowest frequency,highest resolution.

In one embodiment, the layer information indicates the summation of bitsacross the entire image for that layer and all previous layers (forexample the 16,011 bits listed next to layer 1 indicates the totalnumber of bits for layer 0 and layer 1). Alternatively, bytes, words,kilobytes, or other units of memory or rate could be used instead ofbits. Table 7 shows this type of absolute rate information.

Table 8 shows relative rate information. Layer 0 has 4096 bits, layer 1has 11,915 bits, etc.

TABLE 7 layer Rate (bytes) 0 4,096 1 16,011 2 40,000 3 100,000 4 250,0005 500,000 6 1,000,000 7 2,500,000 8 5,500,000

TABLE 8 layer Rate (bytes) 0 4,096 1 11,915 2 23,989 3 60,000 4 150,0005 250,000 6 500,000 7 1,500,000 8 3,000,000

For example, if only 750,000 bytes may be allowed in the decoded image,then all that can be decoded (as the 1,000,000 bytes tabulated withlayer 6 includes the 500,000 bytes of layers 0-5) is through layer 5 andhalf of importance layer 6. In some embodiments, no packets from layer 6would be included. In other embodiments, some packets from layer 6 wouldbe included and others would be replaced by zero packets so that thetotal amount of layer 6 data was approximately 250,000 bytes.

FIG. 22 illustrates an example of layering for a 5,3 irreversibletransform with three levels, MSE or similar. Referring to FIG. 22, thereare 45 layers shown. Each additional layer improves MSE in an order thatgives good rate-distortion for MSE.

FIG. 23 illustrates another example in which transform has 5 levels andthe data is divided up into layers 0-3. Layer 0 corresponds to thethumbnail version, layers 0-1 correspond to the monitor (or screen)resolution, layers 0-2 correspond to the print resolution, and layers0-3 correspond to lossless.

In an alternative embodiment, the layers may be predefined for someother distortion metric (e.g., MSE, weighted MSE, sharpness of text,etc.) The decoder uses the information regarding the layers from thecodestream to select layers to generate an image. The decoder knowingwhat the desired viewing characteristics from the application orimplementation (see Table 9 below), and using the information from thecodestream specifying the layers, can quantize the codestream in orderto display an image at the correct viewing distance. FIG. 9 illustratessuch a decoder. Referring to FIG. 9, decoder 901 receives a codestreamand includes quantization logic 902 that examines the COM marker anduses information about the viewing distance it is at stored in storage903 to generate quantized codestream 904 via, for example, selecting theproper layers. Quantized codestream 904 is decoded by decoding logic 905(e.g., a JPEG 2000 decoder) after selecting layers to generate an imagedata 906. A naive decoder would simply ignore the data in the commentmarker.

FIG. 10 is a flow diagram of a process for using layers when decoding.The process is performed by processing logic that may comprise hardware(e.g., dedicated logic, circuitry, etc.), software (such as is run by,for example, a general purpose computer or a dedicated machine), or acombination of both.

Referring to FIG. 10, the process begins by processing logic receiving acodestream of compressed logic data (processing block 1001). The imagedata is organized into multiple layers, each of which comprises codeddata that adds visual value to the image (e.g., look sharper, betterdefined, better contrast, etc.). Next processing logic selects one ormore layers for quantization based on sideband information (processingblock 1002). After selection, processing logic decompresses thenon-quantized layers of the codestream (processing block 1003).

Editing of Tiles, Tile-parts, or Packets

Once a codestream is created, it may be desirable to edit parts of theimage. That is, for example, after performing encoding to create thecodestream, a set of tiles may be decoded. After decoding the set oftimes, editing may be performed, followed by encoding the set of tileswith the edits to the same size as the encoded tiles were prior to theirdecoding. Examples of typical editing include sharpening of text andremoving “red-eye.” The JPEG 2000 codestream can be edited in memory orin a disk file system without rewriting the entire codestream.

FIG. 11 is a flow diagram of one embodiment of an editing process. Theprocess is performed by process logic that may comprise hardware (e.g.,dedicated logic, circuitry, etc.), software (such as is run by, forexample, a general purpose computer or a dedicated machine), or acombination of both.

Referring to FIG. 11, processing logic initially determines the tiles,tile-parts, or packets that cover the area, resolution, components,and/or precincts to be edited and decodes them (processing block 1101).This determination may be made in response to a user selecting an areaand/or working resolution. The determination may use editing informationfor a higher resolution to determine which parts or tiles cover theportion to be edited. Once decoding has been completed, processing logicperforms the desired edits (processing block 1102).

After performing the desired edits, processing logic recompresses thedata into coded data (processing block 1103) and creates a replacementtile, tile-part, or packet for the codestream (processing block 1104).In one embodiment, in creating the replacement tile, tile-part, orpacket, processing logic pads out the data with bytes at the end of thecodestream if the new data is smaller than the unedited version of thedata to make the replacement tile, tile-part or packet the same size asthe unedited version.

In an alternative embodiment, processing logic may use a marker, or tag,such as a COM marker segment of the appropriate length instead of thepadding. The COM marker could be used to fill space or could containinformation that the encoder wanted to include. It could containinformation such as, for example, sideband information described hereinor a copyright license for an image or text or other file formatinformation.

In one embodiment, in creating the replacement tile, tile-part, orpacket, processing logic truncates the last packets for any or allcomponents until the data fits in the codestream if the new data islarger than the unedited version of the data.

Editing of an image may be performed by changing coded data for tiles,tile-parts, or codeblocks. In one embodiment, editing is performedwithout changing file size by quantizing instead of expanding. Inanother embodiment, a predetermined amount of extra space is allocatedper tile or per codeblock to allow for a predetermined amount ofexpansion. In still another embodiment, coded data may be put at end offiles by manipulating tile headers and putting invalid tile data in COMmarkers.

Note that if there are subsequent tile-parts that depend on the data inthe portion of the codestream that is being edited, these tile-parts maybecome useless in the codestream. An indication of this useless data maybe noted to the decoder by one of several methods. These methods involveinserting or modifying information in the codestream to indicate thepresence and/or location of the useless data. In one embodiment, theapplication uses a status buffer to indicate that the data in tile-partssubsequent to an edited tile-part may be useless. The status buffer maybe in workspace memory and describes dependencies between packets. If anearlier packet is altered, the subsequent packets cannot be decoded asis. These subsequent packets must be edited accordingly or eliminated.In another embodiment, such an indication may be made by zeroing out thedata section of those tile-parts and/or creating a PPT marker segmentthat denotes no data.

Optimal Encoder Quantization

During encoding, unquantized coefficients from some or all subbands maybe divided by a value of Q to create the quantized coefficient values.This value Q may have a wide range of values. Typical encoders quantizea number of the values in a single particular range of values is madeequal to one single coefficient value. In essence, all the coefficientsin the particular range are quantized to the same value. This can beexemplified by FIG. 12 which shows that the range of values is often ina bell shaped curve and that all of the values in the particular range,such as range R₁ are sent to the decoder as one quantized value, such asR₁, and the decoder will reconstruct these values to a particular value.Assume a decoder reconstructs these values to a predetermined value(e.g., floor (½ min+½ max), or min+½ Q, where Q is the quantization stepsize). For example, if the range of values is between 16 and 31, thenthe decoder may assume the value is 24. In one embodiment, instead ofusing ½ as the value, another value is selected, such as floor (⅜ min+⅝max), or min+⅜Q, where Q is the quantization step size. Therefore, ifthe range is from 16 to 31, then it is assumed that the decoder willreconstruct the value to 22, instead of 24.

In some cases, two spatially adjacent coefficients may be close to eachother numerically yet in separate quantization bins, such as coefficientvalues 1201 of range R₂ and 1202 of range R₁ in FIG. 12. The results ofthe quantization may cause an artifact to occur. In one embodiment, forcoefficients near a boundary between two quantization bins, the encoderselects a bin such as Range R₁ into which a coefficient, such ascoefficient 1201, will be quantized so that it is consistent withneighbors, such as coefficient 1202. This helps avoid artifacts. Thatis, this technique reduces distortion yet may increase rate,particularly when a coefficient is moved from a smaller bin to a higherbin.

Flicker Reduction for Motion JPEG

At times, flicker occurs when applying wavelet compression to motionsequences. An example of such flicker may include the image gettingbrighter or darker in areas or the appearance of edges changing insuccessive frames as the motion sequence is played (mosquito noisearound the edges). The flicker may be due to the application ofdifferent local quantization to successive frames of a motion sequenceor to noise exacerbated by quantization that is viewed temporarily.

To reduce flicker, coefficients that are in the same position and closeto the same value in successive frames are forced to the same value.That is, the coefficients values in successive frames are set to apredetermined value. This is essentially a form of quantization that isapplied during encoding. FIG. 13 is a flow diagram of one embodiment ofa process to reduce flicker.

A test of whether to apply such quantization to a coefficient value in asubsequent frame is based on the quantization that was performed on thecoefficient in the previous frame. Thus, the encoder is utilizing framedependency to eliminate flicker while the decoder decodes data frame byframe independently.

In one embodiment, in order to reduce flicker in motion JPEG,coefficient values are modified (quantized) based on their relationshipwith each other with respect to a threshold. For example, if Dn and Dn+1are the corresponding coefficient (same spatial location and samesubband) in two frames before quantization, if D'n and D'n+1 representthese coefficients after quantization, if Q(•) are scalar quantization,and if the value T is a threshold, then the following may be applied:

-   if (|Q(Dn+1)−(D'n)|<T)    -   D'n+1=D'n-   else    -   D'n+1=Q(Dn+1)        For example, the value T may be twice the quantization step        size. Other values of T include, but are not limited to,        √{square root over (2)}Q, 1.5Q, 2√{square root over (2)}Q.

One of the coefficient values may be modified to be either apredetermined closeness to another coefficient value. The closeness maybe determined by some threshold. The threshold may be user set oradaptive based on some criteria. The threshold could be different basedon the subband and, perhaps, on the persistance of the particular value(number of frames that this coefficient is close). In one embodiment,the coefficient value is set equal to the other coefficient value. Inalternative embodiments, the coefficient is set to be within thequantization bin size of the other coefficient value or twice thequantization bin size.

FIG. 14 illustrates one embodiment of an encoder (or portion thereofthat performs the quantization described above. Referring to FIG. 14, aquantizer 1400 receives coefficients 1410 for frames of a motionsequence from a wavelet transform (not shown). The coefficients arereceived by quantization logic 1401 which compares a threshold valuestored in memory 1401 to coefficient values for the previous frame thatare stored in memory 1403 to coefficients 1410 with a scalar quantizer Qapplied from memory 1404.

Quantization logic 1401 may comprise comparison hardware (e.g., logicwith gates, circuitry, etc.) or software to perform the comparison. Thiscomparison hardware and software may implement a subtractor orsubtraction operation. The results are a quantized codesteam (assumingsome values have been changed.) This may be applied over two or moreframes. Also the comparison is not limited to two consecutive frames.The comparison can be over 3, 4, 5, etc., frames, for example, todetermine if a variance exists. FIG. 24 illustrates one example in whichvalues in a first and third frame are used to set the value in thesecond frame.

Note that the quantization can also be codestream quantization with acode block-based rule.

Rate Control, Quantization, and Layering

In one embodiment, selective quantization of coefficients can beperformed during encoding by setting a subset of the refinement bits tobe the more probable symbol (MPS). This may be performed at a userselected bitplane. For examples, if there is text on a background image,with a goal of having sharp text images while minimizing coded datarequired for the background, the refinement bits that are set to MPS arethose that do not effect text for the last bitplane, while using theactual value for bits that effect text.

Such a quantization scheme may be used to implement non-uniformquantization step sizes. For example, if one wanted to have a backgroundwith fewer bits, setting the refinement bits to the MPS could operate asa form of quantization. This quantization scheme causes some level ofdistortion but lowers the bit rate necessary to transfer the codestream.

Note that although this technique may be applied to bits generatedduring the refinement pass, the technique has application to othercompression schemes (e.g., lists generated during subordinate passes,tail bits of CREW of Ricoh Silicon Valley, Menlo Park, Calif., MPEG IVtexture mode, etc.).

In one embodiment, the same technique may be applied to other changesbetween frames. That is, in one embodiment, a change due to a ratedistortion in one frame may be performed in a subsequent frame to avoiddistortion effects.

Rate Control and Quantization

In one embodiment, user specified quantization is provided. For a 3level transform for one component, 7 quantization values are sufficient:level 1 HH, level 1 HL and LH, level 2 HH, level 2 HL and LH, level 3HH, level 3 HL and LH, and level 3 LH.

If quantization values are bitplanes to truncate (which is equivalent toscalar quantization by powers of 2), 3-bit values (0 . . . 7) aresufficient for most applications. (For image components with depth12-bits or more and 5 or more transform levels, perhaps higherquantizations might be useful.) Values 0 . . . 6 could be used tospecify the number of bitplanes to truncate and 7 could be used to meandiscard all bitplanes. The three bit values may be written to acontroller that controls compression (or decompression) hardware (e.g.,JPEG2000 compatible hardware) to perform the quantization.

For 3 component color quantization:

-   -   21 values can be used with separate values for each component,    -   14 values can be used, 7 for luminance and 7 for chrominance,    -   17 values can be used for 4:1:1 subsampled data, 7 for luminance        and 5 for each chrominance component,    -   12 values can be used for 4:1:1 subsampled data, 7 for luminance        and 5 for chrominance,    -   19 values can be used for 4:2:2 subsampled data, 7 for luminance        and 6 for each chrominance component, and    -   13 values can be used for 4:2:2 subsampled data, 7 for luminance        and 6 for chrominance.        Since 21*3=63 bits is less than 8 bytes, transferring or storing        the quantization uses little resources. A central processing        unit (CPU) might select one predetermined quantizer from a table        and write it to a CPU or other controller controlling special        purpose JPEG 2000 hardware (a chip) for each frame of a motion        JPEG 2000 video sequence. Alternatively, one implementation of        JPEG 2000 might have a small memory that holds 8 or 16 different        quantizers that could be selected for each frame.

Quantizers can also be used to assign bitplanes to layers. For example,Q₀, Q₁, and Q₂ may be quantizers that specify bitplanes of coding passto quantize. Quantizer Q₀ causes the most loss, while quantizer Q₂causes the least loss. Layer 1 is all the data quantized by Q₀ but notquantized by Q₁. Layer 2 is all the data quantized by Q₁ but notquantized by Q₂. Layer 3 is all the data quantized by Q₂.

Simple Quantization

FIGS. 17 and 18 show example quantizers (label A . . . Q) for the3-level 5/3 transform as the number of coefficient LSBs to truncate ornot code. Truncating N bitplanes is equivalent to a scalar quantizer of2^(N). The subband where the quantization changes with respect to theprevious quantizer is highlighted with a dashed box. The quantizers D, Kand Q all have the same relationship between the subbands. Otherquantizers might be used that are better for MSE or for other distortionmetrics.

The exemplary Verilog below converts a single quantization value “q”into seven quantizers (number of LSBs to truncate). The variableq_(—)1_HH is used for level 1 HH coefficients, the variable q_(—)1_H isused for level 1 HL and LH coefficients, etc. Some consecutive values ofq result in the same quantizer: 0 and 1; 2 and 3; 4 and 5; 8i+6 and 8i+7for all integers i with i≧0.

module makeQ(q, q_(—)1HH, q_(—)1H, q_(—)2HH, q_(—)2H, q_(—)3HH, q_(—)3H,q_(—)3LL);

-   -   input [5:0] q;    -   output [3:0] q_(—)1HH;    -   output [3:0] q_(—)1H;    -   output [3:0] q_(—)2HH;    -   output [2:0] q_(—)2H;    -   output [2:0] q_(—)3HH;    -   output [2:0] q_(—)3H;    -   output [2:0] q_(—)3LL;    -   wire [3:0] temp_(—)2H;    -   wire [3:0] temp_(—)3HH;    -   wire [3:0] temp_(—)3H;    -   wire [3:0] temp_(—)3LL;    -   wire [2:0] qlo;    -   wire [2:0] qhi;    -   assign qlo=q[2:0];    -   assign qhi=q[5:3];    -   assign q_(—)1HH=qhi+((qlo>=2) ? 1:0);    -   assign q_(—)1H=qhi+((qlo>=4) ? 1:0);    -   assign q_(—)2HH=qhi+((qlo>=6) ? 1:0);    -   assign temp_(—)2H=qhi+((qlo>=1) ? 0:−1);    -   assign temp_(—)3HH=qhi+((qlo>=3) ? 0:−1);    -   assign temp_(—)3H=qhi+((qlo>=5) ? 0:−1);    -   assign temp_(—)3LL=qhi−1    -   assign q_(—)2H=(temp_(—)2H<0) ? 0: temp_(—)2H;    -   assign q_(—)3HH=(temp_(—)3HH<0) ? 0: temp_(—)3HH;    -   assign q_(—)3H=(temp_(—)3H<0) ? 0: temp_(—)3H;    -   assign q_(—)3LL=(temp_(—)3LL<0) ? 0: temp_(—)3LL;

endmodule

Human Visual System Weighting for Color and Frequency

Table 9 shows additional bitplanes to quantize (e.g., truncate) forluminance to take advantage of the frequency response of the HumanVisual System (from Table J-2 of the JPEG 2000 standard). A viewingdistance of 1000 pixels might be appropriate for viewing images on acomputer monitor. Larger viewing distances might be appropriate forprint images or television.

TABLE 9 Human Visual System Weighting for Luminance extra biplanes toquantize for viewing distance of . . . subband 1000 pixels 2000 pixels4000 pixels 1HH 2 4 or 5 discard all 1HL, 1LH 1 2 or 3 6 2HH — 2 4 or 52HL, 2LH — 1 2 or 3 3HH — — 2 3HL, 3LH — — 1Additionally chrominance may be quantized more heavily than luminance.

FIG. 19 shows a quantization that starts with FIG. 17(D) and then addsfrequency weighting for a 1000 pixel viewing distance (to both luminanceand chrominance), keeps 3LL chrominance unchanged, discards 1HL and 1HHchrominance for 4:2:2 and additional 2 bitplanes are discarded for theremaining chrominance.

Sharp text without ringing artifacts is more desirable than exact grayvalue for text/background. That is, if a gray level is supposed to be at50% (for example), and is instead at 60%, it is often not visuallyobjectionable if the image is of text. In one embodiment, the LL (DC)coefficients are quantized more heavily for text than for non-textimages at low bitrate. For example, for an 8-bit image component, aquantiation step size of 8, 16 or 32 might be used for text only regionsand a quantization step size of 1, 2 or 4 might be used for regionscontaining non-text. This allows more fidelity for the high frequencycoefficients, thereby resulting in text with sharp edges.

Using Quantizers to Divide Things into Layers

Table 10 shows 16 example quantizers. Quantizer 15 is lossless.Quantizer 8 is the same as FIG. 19. These can be used divide the subbandbitplanes into layers.

TABLE 10 subband 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Y 1HH all all 6 65 5 4 4 3 3 2 2 1 1 0 0 Y 1HL,LH 6 5 5 4 4 3 3 2 2 1 1 0 0 0 0 0 Y 2HH 54 4 3 3 2 2 1 1 0 0 0 0 0 0 0 Y 2HL,LH 4 4 3 3 2 2 1 1 0 0 0 0 0 0 0 0 Y3HH 4 4 3 3 2 2 1 1 0 0 0 0 0 0 0 0 Y 3HL,LH 4 3 3 2 2 1 1 0 0 0 0 0 0 00 0 Y 3LL 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 C₁ 1HL,HH HL and HH alwaysdiscarded for 4:1:1 or 4:2:2 only C₁ 1LH all all all all 6 6 5 5 4 4 3 32 2 1 0 C₁ 2HH all 6 6 5 5 4 4 3 3 2 2 1 1 0 0 0 C₁ 2HL,LH all 6 5 5 4 43 3 2 2 1 1 0 0 0 0 C₁ 3HH all 6 5 5 4 4 3 3 2 2 1 1 0 0 0 0 C₁ 3HL,LHall 5 5 4 4 3 3 2 2 1 1 0 0 0 0 0 C₁ 3LL all 0 0 0 0 0 0 0 0 0 0 0 0 0 00 C₂ 1HL,HH HL and HH always discarded for 4:1:1 or 4:2:2 only C₂ 1LHall all all all 6 6 5 5 4 4 3 3 2 2 1 0 C₂ 2HH all 6 6 5 5 4 4 3 3 2 2 11 0 0 0 C₂ 2HL,LH all 6 5 5 4 4 3 3 2 2 1 1 0 0 0 0 C₂ 3HH all 6 5 5 4 43 3 2 2 1 1 0 0 0 0 C₂ 3HL,LH all 5 5 4 4 3 3 2 2 1 1 0 0 0 0 0 C₂ 3LLall 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Layer 0 contains all data not quantized away by quantizer 0. This wouldbe luminance data only: all of 3LL; all but 4 bitplanes of 2HL, 2LH,3HL, 3LH and 3HH; all but 5 bitplanes of 2HH and all but 6 bitplanes of1HL and 1LH. Layer 1 contains all data not in layer 0 and not quantizedaway by quantizer 1. This would be luminance bitplanes 5 for 1HL and1LH, bitplane 4 for 2 HH, bitplane 3 for 3HL and 3LH; all 3LLchrominance; all but 5 bitplanes for chrominance 3HL and 31H; and allbut 6 bitplanes for chrominance 2HL, 2LH and 3HH. Finally, layer 15would contain the LSB of 1LH chrominance.

Rate Control with Multiple Layers and Tile-Parts

There several well known techniques for rate control in compressionsystems. The simplist method is to pick a distortion for every “unit”compressed (a unit may be an 8×8 block in JPEG, a frame in a motionsequence, a tile of a single image, a subband of a tile in a waveletcoded image, etc.). If the distortion selected leads to a bitrate higherthan the desired average bitrate, the distortion allowed is increasedfor new units as they are compressed. If the distortion selected leadsto a bit rate lower than the desired average bitrate, the distortionallowed is decreased for new units as they are compressed.

A more complex method buffers the compressed data from some number of“units.” The bitrate and/or distortion for each unit at each distortionlevel is stored. Then the distortion to allow across all the units inthe buffer is determined when the buffer is full. If the buffer issufficient to contain the entire image, extremely high quality resultscan be obtained. In JPEG 2000, layers are designed to contain incrementsto quality. Thus, selecting a distortion can mean selecting the numberof layers to use for each code block or tile. A complete description ofthis type of rate control is in, David Taubman, “High PerformanceScalable Image Compression with EBCOT,” IEEE Transactions on ImageProcessing.

There are several disadvantages to this process. One disadvantage isthat a buffer memory for the entire codestream is required. A seconddisadvantage is that the latency (time until any of the codestream isoutput) is high. A third disadvantage is that the second pass could takelarge amount of time.

To mitigate these problems, each tile of a JPEG 2000 codestream isencoded as described above with at least two layers. At the completionof encoding each tile, a number of packets (e.g., layer, resolution,precinct, tile-component) are output to the codestream as a completetile-part. The remaining layers are stored in the buffer. A second passthrough the remaining coded data in the buffer is optional. During thissecond pass, extra packets from each tile are appended to the codestreamas complete tile-parts as space or time allows. If in a fixed-rateapplication, then only packets within the given rate are appended. If ina fixed time application, then only number of cycles allowed. Oneembodiment of this process is shown in FIG. 15A. Thus, these can be the2 complete tile-parts output for each tile.

FIG. 15B illustrates a number of layers, layers 1-n. Layer 1 is outputon the first pass, and the remaining layers are most likely belowfixed-time or fixed-rate time limits. Layer 2 may be output on a secondpass within fixed-time or fixed-rate requirements while achievingsimilar distortion over all the components.

The above process is advantageous in that it allows the buffer to storea fraction of the coded data, the first data can be output (transmittedor stored) sooner, and the second pass through the data can be fasterbecause there is less data to process. Also less memory is required forbuffering.

The criterion for selecting which packets go into the first set oftile-parts can be similar to any other rate control algorithm. In oneembodiment, the rate of packets can be less than the desired averagebitrate for the whole image. For example, if a final compressedbitstream at 2.0 bpp is desired, the first pass could place 1.5 bpp forevery tile in the codestream, and buffer 1 bpp for every tile.

The second pass can select from the remaining data the packets to placein the second tile part of each tile. Thus, to obtain a 2.0 bpp averageencoding, some tiles that had high distortion after the first pass couldreceive all the remaining data saved for the tile, while other tileparts which had low distortion after the first pass might not have anyadditional data transmitted.

Rate Control for Compressed Codestream Data

Some rate control techniques described herein include rate controlperformed on a compressed codestream based on a request implemented byselecting some number of layers to keep in the codestream. A parser maybe used to produce a new codestream which shows the bitrate based onlayers. This bitrate is equal to or less than the bitrate specified bythe request.

The parser may use a data structure referred to herein as a “packetstructure.” Note that this data structure may be used for other purposessuch as, for example, the versatile pocket data structure describedbelow. In one embodiment, the packet structure includes a packet startpointer and packet length. It also contains a tile number, a resolution,a component, layer, and a precinct the packet belongs to. Finally, italso consists of a selection flag. This flag, when set to apredetermined value (e.g., 1), indicates if the packet is selected inthe array for writing out to a new codestream.

In one embodiment, packets are read in sequential order from acodestream based on the progression order information indicated by theCOD marker.

The number of bytes is computed based on the bitrate desired by therequest. The number of bytes belonging to layer 0 is added up to atotal. Then this total of bytes is compared with the number of bytesdesired. If the total is less than the number of bytes desired, oneadditional layer is added to the total. The process continues until thetotal is equal to or greater than the number of bytes desired or allpackets have been added.

During the process, those packets which have been added to the total,are marked as selected by the selection flag in the structure.

If the total is equal to the number of bytes desired, the additionprocess is stopped. If the total exceeds the number of bytes desired,the packets in the last layer added are subtracted from the total. Thisis done to guarantee that the bitrate is below the bitrate desired.Consequently, during the subtraction step, packets which have beensubtracted from the total are marked unselected.

In one embodiment, the related markers such as SOT, COD, PLT are updatedaccording to the request. Packets are written to the new codestream. Thepacket structure may be created using the following:

typedef struct _PACK_{ /* packet structure */ int start; /* packetstarting point */ int length; /* packet length */ unsigned short t; /*tile number the packet belongs to */ unsigned short r; /* resolution thepacket belongs to */ unsigned short c; /* component the packet belongsto */ unsigned short l; /* layer the packet belongs to */ unsigned shortp; /* precinct the packet belongs to */ unsigned char select;  /*selection flag */ }Pack_t; /* Store packets from tp->tile[i].Size[j]array to the packet structure array *//* Layer progression (LRCP) order*/ if(progression_order == 0){ j = 0; for(i=0;i<number_of_tile;i++){ m =0; for(1=0;<layer;1++){ for(r=0;r<resolution+1;r++){for(c=0;c<component;c++){ for(p=0;p<precinct[r];p++){ tp−>pk[j].start =tp->tile[i].pointer[m]; tp−>pk[j].length = tp->tile[i].Size[m];total_length += tp->tile[i].Size[m]; tp->pk[j].t = i; tp->pk[j].r = r;tp->pk[j].l = l; tp->pk[j].c = c; tp->pk[j].p = p; m++; j++; } } } }num_packet[i] = m; } }Versatile Packet Data Structure

The same packet data structure described above can be used to facilitateother parsing options, once packets are read into the structure.

For resolution parsing, the packets which are to be excluded are markedunselected. For example, given a 4 resolution codestream, and a requestis to produce a 3-resolution codestream, a parser marks all packetswhich belong to resolution 4 unselected. Then the newly producedcodestream contains only packets from resolution 1 up to resolution 3.

Similarly, for component parsing, progression conversion parsing,quality parsing can be performed step by step processing the packets inthe structure.

The packet data structure can handle complex requests. For example, arequest which requires the parser to produce a codestream which has a3-resolution, 2-layer, and 1-component codestream.

Clipping After Each Inverse Transform

As a result of quantization performed on wavelet coefficients, the finaldecoded pixels are often outside of the original range of allowed pixelsfrom the specified bit depth. Typically, these pixels are clipped to theoriginal range so that further image processing or display devices canuse the original bit depth.

For example, an eight bit image has pixel values between 0 and 255,inclusive. After lossy compression is used, the decoded image maycontain values like −5 and 256. To provide an eight bit output, thesevalues are clipped to 0 and 255 respectively. This clipping procedurealways reduces pixel wise distortion because the original image did notcontain pixels outside of the clipping bounds. This procedure is wellknown and recommend by the JPEG 2000 standard.

In addition to the bounds on the final output samples, there are boundson the values coefficients can assume at the various stages of thewavelet transform. Just as quantization can change the final decodedsamples to lie outside the original bounds, quantization can change thepartially transformed wavelet coefficients to lie outside their originalbounds. If these coefficients are clipped to their original bounds,distortion will decrease.

For example, after a horizontal (one dimensional) 5-3 reversibletransform as specified by JPEG 2000 with 8 bit input samples, themaximum value of the low pass coefficient is +191, and the minimumpossible value is −191. The high pass coefficient must be between −255and 255 inclusive. After the vertical one dimensional transform, theLow-Low coefficients are bounded by −286 and 287. Thus when decoding aneight bit image, when the first level low-low pass coefficients aregenerated (by the inverse wavelet transfrom from a higher level), thecoefficients can be clipped to −286 and +287, and distortion willdecrease. Likewise after the first level vertical inverse transformationis done, the low pass coefficients can be clipped to −191,+191, and thehigh pass coefficients can be clipped to −255, 255.

For each subband, each filter, each transform level, and each imagedepth, there is a different maximum and minimum value for thecoefficients. These maximum and minimum values can be computed byfinding the signal that leads to the maximum and minimum and running theforward compression system and recording the maxima. The signals thatlead to extreme values come from inputs where each pixel is either amaximum or minimum. Which pixels should be maximum and which pixelsshould be minimum can be determined by convolving sequences which are −1when the wavelet coefficient is negative and +1 when the waveletcoefficient is negative. For the 5-3 filter used in JPEG 2000 Part I,the low pass signal of interest is [−1+1+1+1−1] and the high pass signalis [−1+1−1].

The signal (image) which will generate the largest LL value is:$\begin{matrix}{+ 1} & {- 1} & {- 1} & {- 1} & {+ 1} \\{- 1} & {+ 1} & {+ 1} & {+ 1} & {- 1} \\{- 1} & {+ 1} & {+ 1} & {+ 1} & {- 1} \\{- 1} & {+ 1} & {+ 1} & {+ 1} & {- 1} \\{+ 1} & {- 1} & {- 1} & {- 1} & {+ 1}\end{matrix}\quad$(where +1 must be replaced by the input maximum (e.g., 255) and −1 mustbe replaced by the input minimum (e.g., 0).

For irreversible filters, it is not necessary to actually run the systemto determine the maxima, simply convolving the wavelet coefficients issufficient. For the reversible 5-3 filter, however, the floor functionis used in the computation of coefficients and is also used to determinethe correct maxima.

Note that this may be used for other filters (e.g., a 9-7 filter).

FIG. 28 is a flow diagram of one embodiment of a process for applying aninverse transform with clipping on partially transformed coefficients.The process is performed by processing logic, which may comprisehardware (e.g., circuitry, dedicated logic, etc.), software (such asthat which runs on a general purpose computer system or a dedicatedmachine), or a combination of both.

Referring to FIG. 28, processing logic applies a first level inversetransform to coefficients (processing block 2801). Thereafter,processing logic clips the partially transformed coefficients to apredetermined range (processing block 2802). Next, processing logicapplies a first level inverse transform to the clipped coefficients(processing block 2803) and clips the partially transformed coefficientsto a predetermined range (processing block 2804), which is differentthan the range in processing block 2802. Again, processing logic appliesa first level inverse transform to clipped coefficients (processingblock 2805) and clips the partially transformed coefficients to stillanother predetermined range (processing block 2806).

Simplified Colorspace Handling

A typical decoding process including color management is shown in FIG.25. Referring to FIG. 25, a file with a file format (e.g., a file formatdescribed in the JPEG 2000 standard) containing a restricted ICC profileis provided to a decoding device. Decompression block 2501 decompressesthe file by taking the codestream portion of the file and performingcontext modeling, entropy decoding, and applying an inverse wavelettransform, but does not perform color space operations. If thecodestream indicates the RCT or ICT component transform should be usedto decode the codestream, these will be performed by block 2502. Thatis, inverse RCT/ICT block 2502 takes the components and the “RCT Y/N”indication (RCT if yes, ICT is no) and performs the specified inversetransform and provides (non-display) RGB pixels. (If specified by thesyntax, inverse level shifting is also performed.)

Finally, the ICC color profile from the file format along withinformation about the display device will be used to produce the outputpixels.

Inverse ICC block 2503 receives the (non-display) RGB pixels and the ICCprofile and applies an inverse color space transform to provide displayRGB pixels.

FIG. 26 illustrates one embodiment of a non-preferred camera encoder.Referring to FIG. 26, a camera generates YCrCb pixels. A converter 2602converts the YCrCb pixels to RGB pixels and provides those two a typicalJPEG 2000 encoder. The encoder comprises a RCT to ICT converter 2603followed by a compressor 2604. The compressor generates an ICC_(A) forcodestream. FIG. 27 illustrates one embodiment of a simpler cameraencoder. That is, instead of including RCT/ICT converter 2603 andcompressor 2604, a simple camera encoder includes only compressor block2702. Referring to FIG. 27, a camera 2701 generates YCrCb pixels andprovides them to compressor 2702. Compressor comprises a JPEG 2000encoder without an RCT conversion and generates an ICC_(B) codestreamwith RCT equaling 1 (with syntax signaling that the inverse RCT shouldbe used on decoding). The relationship between ICC_(B) and ICC_(A) isgiven by the following equation:ICC _(B) =ICC _(A) ∘YCrCb ⁻¹ ∘RCTwhere ∘ represents function composition.

Restricted ICC profiles are “syntaxes” for functions on pixels. A camerawill typically write the same profile for all images, so ICC_(B) iscomputed offline, and copied into each output file. In a prior artsystem there must be HW for YCrCb⁻¹ and RCT/ICT which operates on everypixel.

Coding 4:2:2 and 4:1:1 Data as 4:4:4 Data with Quantization

The JPEG 2000 standard is typically used to handling data in a 4:4:4format. It is not capable of describing how to reconstruct data in 4:1:1or 4:2:2 formats in a 4:4:4 format for output. In one embodiment, whenencoding 4:1:1 data, the encoder treats 1 HL, 1 LH and 1 HH coefficientsas zero. When encoding 4:2:2 data, the encoder treats 1 HL and 1 HHcoefficients as zero. Thus, with all information in the extra subbandsquantized to zero, a decoder is able to receive the codestream in a wayit expects. In other words, the encoded data resembles 4:4:4 data thathas been heavily quantized.

File Order for Thumbnail, Monitor, Printer, and Full Resolution andQuality

Multiple images at multiple resolutions are important in many imageprocessing situations. Depending on the application, a user may want toselect different images of different resolutions. For example, thumbnailimages may be used as an index into a large number of images. Also, ascreen resolution image may be the image used to send to a monitor fordisplay thereon. A print resolution image may be of lower quality forprinter applications.

In one embodiment, a codestream of an image is organized into sectionsso that different versions of the image, such as, for example, athumbnail version, a screen version, a print version and a losslessversion, is progressive by quality.

In one embodiment, the packets are arranged such that certain packetscorrespond to particular resolutions such as a thumbnail. Thecombination of these packets with other packets represents the monitorresolution image, which when combined with other packets may representthe printer version, etc. Using the POC and tile parts, portions of acodestream may be grouped together. For example, all the tiles of thethumbnail size may be grouped together followed by tiles for anotherresolution followed by tiles of another resolution, etc. FIG. 21illustrates an example progression with tile parts for a single server.Each tile's thumbnail is grouped in tile-parts at the beginning of afile. FIG. 21A illustrates that tile-part 2101 is the only portion thatis used for a thumbnail image. FIG. 21B illustrates that for a monitorresolution, tile-parts 2102-2104 have been included with tile-part 2101.FIG. 21C illustrates that for a printer resolution, tile-parts 2105 and2106 have been included with tile-parts 2101-2104. Lastly, FIG. 21Dillustrates that for a lossless version of the data, the remaining threetile-parts 2107-2108 are included with the rest of the tile-parts. Thesesets of tile parts may be placed on a server in this progressive order.

One embodiment of the process for accessing the groupings of tile partsis shown in FIG. 16. The process may be performed by processing logicthat may comprise hardware (e.g., dedicated logic, circuitry, etc.),software (such as is run on a general purpose computer system or adedicated machine), or a combination of both. The following steps assumethat the image has been transformed with sufficient resolution levelsand layers to divide the image into the four sizes.

Referring to FIG. 16, processing logic initially determines the correctresolution and layering for the thumbnail (processing block 1601). Inone embodiment, to determine the correct resolution and layering for thethumbnail, processing logic creates a POC constrained to that resolutionand layer for each tile and then creates a set of tile-parts and placesthis POC for each tile in the codestream.

Next, processing logic repeats processing block 1601 for the monitorresolution given that the thumbnail packets are already in thecodestream (processing block 1602). Then, processing logic repeatsprocessing block 1601 for the printer resolution given that the monitorpackets are already in the codestream (processing block 1603).

Lastly, processing logic creates a POC marker with the extremes of theresolutions and layers for each tile (processing block 1604). In oneembodiment, creating the POC with the extremes of the resolutions andlayers is performed by creating a fourth set of tile-parts with theremaining tile-parts for a lossless version.

Note that the particular orders of the packets defined in the POCs arenot of importance, only the limits.

An Exemplary Computer System

FIG. 20 is a block diagram of an exemplary computer system. Referring toFIG. 20, computer system 2000 may comprise an exemplary client 150 orserver 100 computer system. Computer system 2000 comprises acommunication mechanism or bus 2011 for communicating information, and aprocessor 2012 coupled with bus 2011 for processing information.Processor 2012 includes a microprocessor, but is not limited to amicroprocessor, such as, for example, Pentium™, PowerPC™, Alpha™, etc.

System 2000 further comprises a random access memory (RAM), or otherdynamic storage device 2004 (referred to as main memory) coupled to bus2011 for storing information and instructions to be executed byprocessor 2012. Main memory 2004 also may be used for storing temporaryvariables or other intermediate information during execution ofinstructions by processor 2012.

Computer system 2000 also comprises a read only memory (ROM) and/orother static storage device 2006 coupled to bus 2011 for storing staticinformation and instructions for processor 2012, and a data storagedevice 2007, such as a magnetic disk or optical disk and itscorresponding disk drive. Data storage device 2007 is coupled to bus2011 for storing information and instructions.

Computer system 2000 may further be coupled to a display device 2021,such as a cathode ray tube (CRT) or liquid crystal display (LCD),coupled to bus 2011 for displaying information to a computer user. Analphanumeric input device 2022, including alphanumeric and other keys,may also be coupled to bus 2011 for communicating information andcommand selections to processor 2012. An additional user input device iscursor control 2023, such as a mouse, trackball, trackpad, stylus, orcursor direction keys, coupled to bus 2011 for communicating directioninformation and command selections to processor 2012, and forcontrolling cursor movement on display 2021.

Another device that may be coupled to bus 2011 is hard copy device 2024,which may be used for printing instructions, data, or other informationon a medium such as paper, film, or similar types of media. Furthermore,a sound recording and playback device, such as a speaker and/ormicrophone may optionally be coupled to bus 2011 for audio interfacingwith computer system 2000. Another device that may be coupled to bus2011 is a wired/wireless communication capability 2025 to communicationto a phone or handheld palm device.

Note that any or all of the components of system 2000 and associatedhardware may be used in the present invention. However, it can beappreciated that other configurations of the computer system may includesome or all of the devices.

Whereas many alterations and modifications of the present invention willno doubt become apparent to a person of ordinary skill in the art afterhaving read the foregoing description, it is to be understood that anyparticular embodiment shown and described by way of illustration is inno way intended to be considered limiting. Therefore, references todetails of various embodiments are not intended to limit the scope ofthe claims which in themselves recite only those features regarded asessential to the invention.

1. A system comprising: a memory sized to include lines to store a bandof an image and additional lines; a wavelet processing logic comprisinga wavelet transform to generate coefficients when applied to data in thememory; access logic to read data from the memory into line buffers tosupply data stored in the memory to the wavelet transform and to storecoefficients in the memory, such that after data stored at a first pairof lines is read from memory into the line buffers of the access logic,the access logic reuses the first pair of lines to store coefficientsgenerated by the wavelet transform that are associated with a secondpair of lines different from the first pair of lines.
 2. The systemdefined in claim 1 wherein the access logic stores coefficients incontiguous lines of memory with coefficients from a same subband anddecomposition level adjacent to each other.
 3. The system defined inclaim 1 wherein a first line of each of the first and second pairs oflines are located in the memory at an offset with respect to each other.4. The system defined in claim 3 wherein the wavelet transform comprisesa plurality of transform levels and coefficient levels, and wherein theaccess logic stores first outputs of the wavelet transform for eachcoefficient level in additional lines within a distance of the offset.5. The system defined in claim 4 wherein size of the offset is differentfor each transform level.
 6. The system defined in claim 5 wherein thesize of the offset is equal to:2^((transform level of coefficient being stored)).
 7. The system definedin claim 6 wherein, during decomposition, the offset for storing a firstrow of rows of L1 coefficients in the memory is two lines from the firstrow of data of an image associated with the L1 coefficients, and theoffset for storing a first row of L2 coefficients is four lines from thefirst row of L1 coefficients associated with the L2 coefficients.
 8. Thesystem defined in claim 1 wherein the access logic stores coefficientsassociated with a decomposition level greater than level three in thelines of the memory that previously stored the band of the image.
 9. Thesystem defined in claim 3 wherein additional lines relating to theoffset are above a line storing the band of the image.
 10. The systemdefined in claim 1 wherein the wavelet transform is a forward wavelettransform.
 11. The system defined in claim 1 wherein the wavelettransform is an inverse wavelet transform.
 12. A method comprising:reading data from a memory into line buffers to apply a wavelettransform thereto; and storing coefficients created by applying thewavelet transform at lines in the memory, including access logic reusinga first pair of lines to store coefficients generated by a wavelettransform, that are associated with a second pair of lines differentfrom the first pair of lines, after data stored at a first pair of linesis read from memory into the line buffers of the access logic, andwherein a first line of each of the first and second pairs of lines arelocated in the memory at an offset with respect to each other.
 13. Themethod defined in claim 12 wherein the wavelet transform includes aplurality of transform levels and coefficient levels, and wherein themethod further comprises the access logic storing first outputs of thewavelet transform for each coefficient level in additional lines withina distance of the offset.
 14. The method defined in claim 13 whereinsize of the offset is different for each transform level.
 15. The methoddefined in claim 14 wherein the size of the offset is equal to:2^((transform level of coefficient being stored)).
 16. The methoddefined in claim 15 wherein, during decomposition, the offset forstoring a first row of L1 coefficients in the memory is two lines fromthe first row of data of an image associated with the L1 coefficients,and the offset for storing a first row of L2 coefficients is four linesfrom the first row of L1 coefficients associated with the L2coefficients.
 17. The method defined in claim 12 further comprisingaccess logic storing coefficients associated with a decomposition levelgreater than level three in the lines of the memory that previouslystored a band of an image.
 18. The method defined in claim 17 whereinadditional lines relating to the offset are above a line storing theband of the image.
 19. An article of manufacture comprising at least onerecordable media storing executable instructions thereon which, whenexecuted by a processing device, cause the processing device to: readdata from a memory into line buffers to apply a wavelet transformthereto; and store coefficients created by applying the wavelettransform at lines in the memory when instructions to cause theprocessing device to store coefficients includes instructions which whenexecuted cause the processing device to reuse a first pair of lines tostore coefficients generated by a wavelet transform, that are associatedwith a second pair of lines different from the first pair of lines,after data stored at a first pair of lines is read from memory into theline buffers of access logic, and wherein a first line of each of thefirst and second pairs of lines are located in the memory at an offsetwith respect to each other.
 20. An apparatus comprising: means forreading data from a memory into line buffers to apply a wavelettransform thereto; and means for storing coefficients created byapplying the wavelet transform at lines in the memory, wherein the meansfor storing includes means for reusing a first pair of lines to storecoefficients generated by a wavelet transform, that are associated witha second pair of lines different from the first pair of lines, afterdata stored at a first pair of lines is read from memory into the linebuffers of access logic, and wherein a first line of each of the firstand second pairs of lines are located in the memory at an offset withrespect to each other.
 21. The article of manufacture defined in claim19 wherein the wavelet transform includes a plurality of transformlevels and coefficient levels, and wherein the article of manufacturefurther comprises the access logic storing first outputs of the wavelettransform for each coefficient level in additional lines within adistance of the offset.
 22. The article of manufacture defined in claim21 wherein size of the offset is different for each transform level. 23.The article of manufacture defined in claim 22 wherein the size of theoffset is equal to:2^((transform level of coefficient being stored)).
 24. The article ofmanufacture defined in claim 21 wherein, during decomposition, theoffset for storing a first row of L1 coefficients in the memory is twolines from the first row of data of an image associated with the L1coefficients, and the offset for storing a first row of L2 coefficientsis four lines from the first row of L1 coefficients associated with theL2 coefficients.
 25. The article of manufacture defined in claim 19wherein the method further comprises access logic storing coefficientsassociated with a decomposition level greater than level three in thelines of the memory that previously stored a band of an image.
 26. Thearticle of manufacture defined in claim 25 wherein additional linesrelating to the offset are above a line storing the band of the image.27. The apparatus defined in claim 20 wherein the wavelet transformincludes a plurality of transform levels and coefficient levels, andwherein the apparatus further comprises means for the access logicstoring first outputs of the wavelet transform for each coefficientlevel in additional lines within a distance of the offset.
 28. Theapparatus defined in claim 27 wherein size of the offset is differentfor each transform level.
 29. The apparatus defined in claim 28 whereinthe size of the offset is equal to:2^((transform level of coefficient being stored)).
 30. The apparatusdefined in claim 27 wherein, during decomposition, the offset forstoring a first row of L1 coefficients in the memory is two lines fromthe first row of data of an image associated with the L1 coefficients,and the offset for storing a first row of L2 coefficients is four linesfrom the first row of L1 coefficients associated with the L2coefficients.
 31. The apparatus defined in claim 20 further comprisingmeans for access logic storing coefficients associated with adecomposition level greater than level three in the lines of the memorythat previously stored a band of an image.
 32. The apparatus defined inclaim 31 wherein additional lines relating to the offset are above aline storing the band of the image.